This adds an adapter to integrate with SolidQueue, reporting job queue time and busy metrics (if enabled) to Judoscale for autoscaling.

SolidQueue is currently on v0.3, still pretty early on, and there's still some things being figured out, but there's early adoption and we expect more as it becomes a Rails recommendation / default in the future. It only works with Rails v7.1+ and Ruby 2.7+, so that's what this adapter will support initially.

We'll be collecting queue time / latency via the "ready executions" table, and busy via the "claimed executions" table.

SolidQueue moves jobs between different tables as they change "status", in other words, while all jobs have a representation on the main "jobs" table, they also get a record on an associated table that may represent what's happening to them: when they're ready to be picked up for work, they go to "ready executions", when they're claimed by a process worker to be performed, they go to "claimed executions", and if there's a failure (that's not retired by Active Job), they go to "failed executions"; if they're scheduled to run in the future, they go to "scheduled executions" (or if they're being retried by AJ, which is essentially re-scheduling them in the future, until it succeeds or gives up retrying and blows up back to SolidQueue.)

When jobs are finished successfully, they are flagged with a "finished_at" column on the main "jobs" table. As the jobs moves from one to the other "execution" status in the workflow, their previous record is destroyed, so there should be really only one of those "execution" representations at one point in time. (i.e. a job is either scheduled, ready, claimed, failed)

There's also the concept of recurring executions, which are created via config (a cron-like setup), and eventually get added to "ready executions" for every recur.

And finally, there's one thing I have to look a bit more: blocked executions. It seems you can add concurrency limit to jobs, which may lock certain jobs from running (if they're concurrency limited by a certain condition) and will move them to a separate "blocked executions" table. I would like to test this more, because I'm wondering if we need to check this table for jobs in order to calculate queue time as well.

Todo / Questions

• Investigate "blocked executions" / concurrency limits, to determine whether they should be added to the queue time / latency.
  • I've been playing with this some, and it works as you'd expect: you can setup a job with a concurrency limit, i.e. run only one job at a time, or one job with this set of arguments, or up to X jobs concurrently, etc., and if more jobs are enqueued, instead of going to "ready", they go to "blocked". When jobs are finished, they check for blocked jobs to unblock them, and there's also an additional dispatcher that checks for blocked jobs on a schedule.
  • While initially I thought it'd make sense to consider these for the latency calculation, the more I thought about and played with it, the more it came to mind that having a big list of blocked jobs doesn't mean a need to autoscale: you might simply be limiting the concurrency of those jobs to a point where many are getting enqueued at certain points, but just a few get processed due to the limits imposed. This could cause the blocked execution table to grow temporarily, causing those blocked jobs to have "increased latency", but autoscaling up might be wrong in this case, since more processing power won't make those jobs complete any faster -- they're still limited by their concurrency setup. In other words, I'm thinking that autoscaling should only look for jobs in the "ready execution" table initially.

          ::SolidQueue::Job
            .left_joins(:blocked_execution, :ready_execution)
            .merge(::SolidQueue::BlockedExecution.where.not({ id: nil }))
            .or(::SolidQueue::ReadyExecution.where.not({ id: nil }))
            .group(:queue_name)
            .minimum("coalesce(#{::SolidQueue::BlockedExecution.table_name}.created_at, #{::SolidQueue::ReadyExecution.table_name}.created_at)")